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ABSTRACT 



The Schools and Staffing Survey (SASS) of the National 
Center for Education Statistics offers the most comprehensive picture 
available of elementary and secondary schools in the United States. The value 
of the SASS would be even greater if information on districts and schools 
could be related directly to student outcomes. A relatively inexpensive 
source of school-level achievement data is state assessment data. Through 
cooperation with the states that have such data, it is possible to develop an 
SASS student achievement subfile that adds an outcome dimension to the SASS 
and provides in-depth data for use by states in understanding the 
organizational factors that are associated with variation in their schools* 
achievement test scores. If a linkage were carried out in all possible 
states, state assessment data would be missing for only about one- third of 
the SASS public schools. To address questions about feasibility, power, 
accuracy, and generalizability of analyses that combine SASS data with state 
assessment data, a pilot study was undertaken that involved merging 1993-94 
SASS data with state assessment data from selected states. Chapter 2 of this 
report describes the development of the pilot project, and chapter 3 presents 
results concerning the developed SASS student achievement measure and 
preliminary analyses of a model relating student achievement levels to 
student background characteristics and school attributes measured by the 
SASS. An appendix presents technical notes. (Contains 4 figures, 11 tables, 
and 27 references.) (SLD) 
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1 . Purpose and Description of SASS Student 
Assessment Subfile Pilot Study 



The Schools and Staffing Survey (SASS), conducted by the National Center for 
Education Statistics (NCES), offers the most comprehensive picture available of 
elementary and secondary schools in the United States. Initiated in 1987-88 and repeated 
in 1990-91 and 1993-94, SASS consists of surveys of districts, principals, and teachers 
associated with a national sample of schools. Through its four core components — ^the 
Teacher Demand and Shortage Questionnaire for Public School Districts, the School 
Questionnaire, the School Principal Questionnaire, and the Teacher Questionnaire — 
SASS provides information on: 

• district enrollment and staffing patterns (for public schools), along with 
district policies, programs, and services 

• school characteristics and staffing, as well as school policies, programs, and 
services 

• principals’ backgroimds, education, and experience, together with their 
perceptions of school decision-making procedures and problems 

• teachers’ training, experience, and compensation, along with their attitudes 
toward teaching and their future plans 

Given the broad reach of SASS, it can speak to a variety of important educational 
research and policy questions. The value of SASS would be even greater, however, if 
information on districts and schools could be directly related to student outcomes. With 
such data available, SASS could more meaningfully inform debates over the factors that 



relate to school effectiveness and contribute to a broad-based evaluation of school 
improvement strategies. Boruch and Terhanian (1996) have pointed out the added value 
of linking NCES surveys to data from other sources. 

A relatively inexpensive source of school-level achievement data, more accessible 
now than ever before, is state assessment data. Most states are currently involved in some 
form of assessment of school performance in terms of student test scores.* Through 
cooperation with these states, it is possible to develop a SASS student achievement subfile 
that both adds an outcome dimension to SASS and provides in-depth data for use by states 
in understanding the organizational factors that are associated with variation in their 
schools’ achievement test scores. 

Of course, state assessment data are only available for public SASS schools; and 
some SASS schools are located in states that do not carry out student assessments. If states 
that conduct assessments differ in important ways from those that do not, then results of 
analyses based on a SASS student achievement subfile might not be entirely generalizable 
to public schools nationwide. Also, state assessments are only conducted in some grade 
levels, and some SASS public schools (e.g., many high schools) may have no students 
enrolled in those grades. Nonetheless, if a linkage were carried out in all possible states, 
state assessment data would be missing for only about one-third of the 8,767 SASS public 
schools. Moreover, because the public school sampling design for SASS uses “state” as a 
major stratification variable, analyses restricted to the subset of states for which 
assessment data are available would be valid and meaningful. 

Pooling the results of analyses of individual state data across states adds substantial 
power to the comparison of SASS organizational measures to achievement measures. 
However, this pooling would be facilitated by placing the state assessment data on a 
common scale. An important aspect of this study is to determine whether the availability 
of State NAEP data for most states that conduct state assessments will support the creation 
of a common achievement metric. Making use of State NAEP data, linkages can be 
constructed that allow the transformation of different state assessments with a similar 
focus (e.g., reading) onto a common metric. Although not essential for the use of state 
assessment data with SASS, such linkages will allow between-state variation to be 
incorporated into school-level achievement scores. 

To address questions about the feasibility, power, accuracy, and generalizability of 
analyses that combine SASS data with state assessment data, ESSI imdertook this pilot 
study, which involves merging 1993-94 SASS data with state assessment data from 
selected states. Chapter 2 of this report describes the development of the pilot SASS 

‘ Forty-five states had statewide assessment systems in 1994-95; the remaining five states either did not have 
a statewide system at all or had temporarily suspended their programs (National Education Goals Panel 
1996). In 1995-96, forty-six states administered statewide assessments (Roeber, Bond, and Braskamp, 

1996) . 

^ A separate project is being carried out by ESSI to evaluate the feasibility of linking state assessments to 
NAEP, using student-level data. The SASS student achievement subfile pilot study, reported here, is not 
directly related to that other linking study. 



student achievement subfile and the preparation of the different measmes of student 
achievement used in the study. Chapter 3 then presents analytical results concerning the 
developed SASS student achievement measiue and preliminary analyses of a model 
relating student achievement levels to student background characteristics and school 
attributes measured by SASS. 



2. Development of a SASS Student 
Achievement Subfile 



This chapter describes (1) the principal data sets used in linking SASS and state 
assessment data, (2) the procedures followed in merging school information from SASS 
with school information from state assessment files, (3) the results of the merging process, 
and (4) the development of an achievement measure for inclusion on the SASS student 
achievement subfile. School-level state assessment data are reported in a different format 
in each state, as they are not intended to be aggregated with assessment data from other 
states. As a result, some effort is involved in creating SASS-state assessment linkages. 
The details of the linkage process are documented here to provide a basis for similar file 
creation in the future; however, readers who wish primarily to evaluate the utility of the 
file thus created may skip the remainder of this chapter and proceed to Chapter 3. 

We did not expect to be able to add an achievement measure for every SASS 
public school, because some schools are not assessed, and a few states had no systematic 
achievement assessment program in 1993-94. However, the aim was to add a measure for 
most SASS public schools in each of 19 selected states and to identify the reasons why 
assessment scores were not available for the remainder of the schools in the 19 states. 
With these results, it becomes possible to estimate the scope and generalizability of 
findings based on the SASS student achievement subfile. 



Data Sets Used in Linking SASS and State Assessment Data 

Both federal and state data sets were used to conduct this study: the 1993-1994 
Schools and Staffing Survey (SASS) data file, the 1991-1992 Common Core of Data 
(CCD) file, nineteen 1993-1994 state assessment data files (obtained and cleaned as part 
of a previous project), and the 1 994 State NAEP fourth grade reading data file. 

SASS and CCD 

The 1993-94 SASS and 1991-1992 CCD files were obtained fromNCES. The 
1991-92 Co mm on Core of Data formed the public school sampling frame for 1993-94 
SASS.^ A comprehensive national database maintained and updated annually by NCES, 
CCD provides descriptive and fiscal information on schools and districts, along with data 
on staff and students. 

A critical feature (not widely known) of the CCD files is that for most public 
schools in the nation they contain both standard federal identification codes and the codes 
used by individual states to identify the schools on state data files. The presence of these 
two identification codes, one of which matched the SASS school identification code (on 
the restricted use data set) and the other of which matched the school identification code 
on the state assessment data file, facilitated the matching process in most (but not all) 
cases. SASS data were linked to CCD by way of the standard federal identification codes 
for schools (present on both SASS and CCD files), and the resulting records were linked 
to state assessment data by way of the school identifiers used by the states (present on both 
CCD and state assessment files). Many of the SASS schools in the states included in the 
study for which state assessment data were not available were very small schools or 
schools serving special populations of students. 

State assessment data 

The state assessment data used for the current study were obtained from an existing 
database maintained at the American Institutes for Research (AIR). For a separate study of 
the representativeness of the State NAEP sample, funded by the NAEP Data Reporting 
Grant program, AIR had previously contacted the thirty-nine states that participated in the 
1994 State NAEP fourth grade reading assessment and had succeeded in gathering school- 
level state assessment scores from twenty-three of them. Assessment data from four of 
these states were deemed inappropriate for use in the current pilot study, leaving a total of 
nineteen state assessment data sets.** These 19 states contained 3,668 of the 8,767 SASS 
public schools. 



^ “The CCD is based on survey data collected annually by NCES from all state education agencies, and is 
believed to be the most complete list of public schools available. The frame includes regular public schools, 
Department of Defense-operated military base schools, and noiuegular schools such as special education, 
vocational, and alternative schools” (Henke, Choy, and Geis, 1996, p. 193). 

* The nineteen states included in the present study were: Alabama, California, Delaware, Florida, Georgia, 
Hawaii, Kentucky, Louisiana, Maine, Massachusetts, Michigan, Montana, New York, Pennsylvania, Rhode 
Island, Tennessee, Texas, Utah, and Washington. Arkansas was excluded because reading assessment 



State NAEP data 



The 1994 State NAEP fourth grade reading data file was acquired from the 
Educational Testing Service (ETS), with NCES approval, for use in exploring the value of 
adjusting state assessment scores to account for between-state variation. 



Merge Procedures 

Three steps were involved in matching state assessment scores with SASS public 
schools. 

Step 1: Selection of SASS schools with students enrolled in assessed grades 

Reading assessment scores from the 1993-94 school year were used in this study, 
both because of the importance of reading as a measure of educational achievement and 
because of the availability of NAEP fourth-grade reading scores for the same year. Table 1 
lists the grade levels for which state reading assessment data were available in the nineteen 
states included in this study. All nineteen states had assessments at the elementary level, 
and sixteen states also had assessments at junior high or high school levels. SASS schools 
that did not report students enrolled in at least one grade covered by assessments in their 
states were eliminated before any attempt was made to merge with state assessment files. 

A total of 2,859 SASS public schools in the nineteen states enrolled students in a grade 
that was assessed in the corresponding state assessment.^ 



scores were reported at the district level rather than the school level; Wisconsin was excluded because data 
were available only for a min imum competency test; and Connecticut and New Hampshire were excluded 
because of pilot study limits on time for data entry and matching. 

^ Some schools enrolled students in more than one grade covered by assessments in their states. Where one 
of these grades was grade 4, we used the fourth-grade assessment scores. Otherwise, we used assessment 
scores associated with the lowest of the covered grades. In 18 of the 19 states, essentially all regular schools 
participated in state assessments. In Pennsylvania, however, only one-third of schools were included in 
1994, as a random sample. 



Table 1 — Number of SASS public schools in 19 states with state assessment data 



State 


Grades with 
assessment data 


# of SASS 
schools 


# with assessed 
grades 


# without assessed 
grades 


Alabama 


4,8, - 


224 


160 


64 


California 


4, 8, 10 


352 


311 


41 


Delaware 


3, 8, 10 


63 


55 


8 


Florida 


4,8, - 


228 


166 


62 


Georgia 


3,5, 8, 11 


168 


164 


4 


Hawaii 


3,8, - 


85 


77 


8 


Kentucky 


4, 8, 12 


149 


140 


9 


Louisiana 


4, 7, 11 


207 


189 


18 


Maine 


4, 8, 11 


145 


136 


9 


Massachusetts 


4,8, - 


208 


116 


92 


Michigan 


4, -, - 


202 


78 


124 


Montana 


3,4, 8, 11 


178 


172 


6 


New York 


3, -, - 


269 


88 


181 


Pennsylvania 


5, 8, 11 


169 


148 


21 


Rhode Island 


4, 8, 10 


88 


81 


7 


Tennessee 


4,8, - 


179 


115 


64 


Texas 


4, 8, 10 


380 


355 


25 


Utah 


5, 8, 1 1 


174 


163 


11 


Washington 


4,8, - 


200 


145 


55 


Totals 




3,668 


2,859 


809 



Step 2: Verification of SASS schools 

The CCD file provided an important link between SASS and the state assessment 
data. Given that a CCD public school identifier was available on the restricted SASS file,^ 
and that most CCD records contained codes used by states to identify their schools, SASS 
schools could be matched to state assessment data if SASS data could first be matched to 
CCD data. Four CCD variables were then used to match schools to state assessment files: 
CCD state agency ID, CCD school agency ID, school name, and city. Table 2 shows, 
however, that in the 19 states included in this study a total of 214 SASS public schools 
with positive enrollments in grades covered by their state assessments did not have valid 
1991-92 CCD school identifiers on the SASS file (most were blank or nines). Fortunately, 
NCES was able to provide a separate file containing the names and addresses of these 
SASS schools to help in matching them to state assessment data. 



^ The variable was not on the restricted SASS file at the outset of the study, but a separate file was provided 
to AIR with the CCD identifiers. 



Table 2 — Number of SASS schools with assessed grades, by CCD identifier 



State 


SASS schools 
with assessed 
grade 


SASS schools with 
assessed grade and a 
valid CCD identifier 


SASS schools with 
assessed grade but not 
a valid CCD identifier 


Alabama 


160 


149 


11 


California 


311 


289 


22 


Delaware 


55 


52 


3 


Florida 


166 


141 


25 


Georgia 


164 


145 


19 


Hawaii 


77 


76 


1 


Kentucky 


140 


135 


5 


Louisiana 


189 


180 


9 


Maine 


136 


128 


8 


Massachusetts 


116 


109 


7 


Michigan 


78 


69 


9 


Montana 


172 


143 


29 


New York 


88 


87 


1 


Pennsylvania 


148 


138 


10 


Rhode Island 


81 


74 


7 


Tennessee 


115 


111 


4 


Texas 


355 


331 


24 


Utah 


163 


158 


5 


Washington 


145 


130 


15 


Total 


2,859 


2,645 


214 



Step 3: Matches to State Assessment Files 

State identification codes on the CCD file and school and city names were used to 
merge SASS schools with state assessment schools.’ State assessment files in sixteen 
states had school codes, most of which could be matched to substrings of the CCD state 
agency ID and the CCD school agency ID.* Montana only had district data and codes; and 
for Alabama, we used city name and school name to merge SASS schools with state 
assessment schools. For Rhode Island, scores were entered from a printed report, after a 
match on school and district/city name. Secondary merges were employed for SASS 
schools that did not initially match with state assessment schools, due to changes in school 
codes, school names, etc. School names and district names were used as secondary 
variables for verification and additional merges in all states except Louisiana and New 
York, where these variables were not present on state assessment files. 



’ See Section 3 of Appendix A for a table showing the specific CCD fields used for merging with state 
assessment data for each of the nineteen states. 

‘ State agencies’ codes varied in length, but the CCD fields for the state education agencies’ school 
identification codes were allocated a length sufficient to include the longest of the state sets of codes. 



Merge Results 



^ Table 3 provides an overview of the merge results. Of the 3,668 SASS public 

schools in the 19 selected states, 2,859 had students enrolled in grades corresponding to 
the state assessment. We successfully merged 2,575 of these SASS schools with state 
assessment scores. These schools enroll 40 percent of the nation’s public school students. 
Of the other 284 SASS schools, 254 did not match with state assessments, and 30 merged 
^ with state assessment files but did not have valid reading scores. The 30 SASS schools 

without valid reading scores included 22 in California, 1 in Florida, 3 in Maine, and 4 in 
Texas. 



Overall, we were able to merge all the SASS public schools with assessed grades 
to state assessment schools in four of the nineteen states: Alabama, Hawaii, New York, 
and Tennessee. Eight other states had fewer than 10 SASS public schools with assessed 
grades that did not match schools on the state assessment data file. 

Table 3 — Results of merging SASS schools with state assessments 



State 


No. of SASS 
schools with an 
assessment grade 


No. of SASS 
schools with state 
assessment scores 


No. of SASS schools 
on state assessment file, 
but missing scores 


No. of SASS 
schools with assessed 
grade not matched 


Alabama 


160 


160 


0 


0 


California 


311 


240 


22 


49 


Delaware 


55 


50 


0 


5 


Florida 


166 


140 


1 


25 


Georgia 


164 


163 


0 


1 


Hawaii 


77 


77 


0 


0 


Kentucky 


140 


133 


0 


7 


Louisiana 


189 


184 


0 


5 


Maine 


136 


123 


0 


13 


Massachusetts 


116 


110 


3 


3 


Michigan 


78 


76 


0 


2 


Montana 


172 


160 


0 


12 


New York 


88 


88 


0 


0 


Pennsylvania 


148 


55 


0 


93 


Rhode Island 


81 


80 


0 


1 


Tennessee 


115 


115 


0 


0 


Texas 


355 


335 


4 


16 


Utah 


163 


160 


0 


3 


Washington 


145 


126 


0 


19 


Total 


2,859 


2,575 


30 


254 



Table 4 lists the number of SASS public schools that did not match with state 
assessment schools, broken down into special, alternative, or vocational education schools 
(e.g. Special Ed Center, Alternative Center etc.) and schools that had enrollment of less 
than 10 in the grade assessed by the state. Of the 254 SASS public schools that did not 
• match with state assessment files, 112 were special, alternative, or vocational education 

schools, or schools that had an enrollment of fewer than 10 in the grade assessed. The 



ERIC 



9 



21 



large number of special schools in California, Florida, Maine, and Texas are responsible 
for many of the non-matches between SASS schools and state assessment files. 

Other than the special and very small schools in those 19 states, only 142 public 
schools in SASS with enrollments in assessed grades did not match with state 
assessments. Most of these were in three states: Pennsylvania, California, and 
Washington. Of the 142 schools, 86 were in Pennsylvania, where matrix sampling was 
used and only one-third of the schools in the state participated in the 1994 state 
assessment.^ In California, the state assessment system had undergone political problems, 
and this may be the reason we did not find 21 SASS schools on the state assessment file. 
These political problems may also help explain why 22 other SASS schools that were 
found on the California state assessment file did not have valid reading scores (see Table 
3). Corresponding explanations for the 36 remaining non-matched schools in 10 other 
states were not sought in detail. However, some were high schools that had very small, 
but nonzero, 8* grade enrollments. It is understandable diat those students might not be 
included in a state assessment. 

To summarize, of the 3,668 SASS schools in the 19 states included in this study, 
assessment scores were found for 2,575. Of the remaining schools, 809 had no students in 
the grades covered by state assessments; 81 were special, alternative, or vocational 
schools; 3 1 had fewer than 10 students in the assessed grades; 30 did not have valid 
reading scores on the state assessment files; and 86 were located in Pennsylvania, where 
only a sample of schools was assessed in 1994. Thus, average achievement scores were 
identified for 2,575 of the 2,631 SASS schools that were expected to have matching 
assessment scores, for a match rate of 97.9 percent. Improvement on this match rate 
would require calls to individual state agencies, to resolve matching problems created by 
school name changes, etc. 

The grade level distribution of schools included on the pilot version of the SASS 
student achievement subfile differs somewhat from that of the entire range of SASS public 
schools, because many states only conduct assessments at the elementary and intermediate 
levels. As shown in Table 5, the population (weighted) percentage of schools with 
average student achievement scores that were high schools is only about half as great as 
the overall percentage of high schools (1 1.5 percent, vs. 23.9 percent), and none of the 
small munber (1.4 percent) of ungraded schools were included. For this reason, it is 
essential that analyses of the SASS student achievement subfile be carried out within 
grade level; analyses that lump all grade levels together would over-represent elementary 
schools. 



® We assume that nearly all of the 86 non-matching Pennsylvania schools were not included in the state’s 
assessment sample in 1994, but it is possible that some number may be true non-matches (i.e., SASS schools 
for which SASS and state assessment information could simply not be merged). In 1995-96, by comparison, 
Pennsylvania assessed all 5*, 8*, and 1 1* grade students in reading (Roeber, Bond, and Braskamp, 1996). 



Table 4 — Number of SASS schools that did not match with state assessment, by state 



Number of SASS public schools with enrollment in the assessed grade but not matched 







Special, alternative. 


Schools with 






Total 


or vocational 


enrollment less than 10 


Other schools 


State 




education schools 


in assessed grade 




Alabama 


0 


0 


0 




0 


California 


49 


21 


7 




21 


Delaware 


5 


2 


3 




0 


Florida 


25 


16 


6 




3 


Georgia 


1 


0 


0 




1 


Hawaii 


0 


0 


0 




0 


Kentucky 


7 


3 


1 




3 


Louisiana 


5 


5 


0 




0 


Maine 


13 


11 


2 




0 


Massachusetts 


3 


0 


2 




1 


Michigan 


2 


0 


1 




1 


Montana 


12 


1 


3 




8 


New Yoric 


0 


0 


0 




0 


Pennsylvania 


93 


6 


1 




86 


Rhode Island 


1 


0 


0 




1 


Tennessee 


0 


0 


0 




0 


Texas 


16 


11 


2 




3 


Utah 


3 


0 


2 




1 


Washington 


19 


5 


1 




13 


Total 


254 


81 


31 




142 


Table 5 — Weighted distribution of public schools with achievement scores and other public schools, 


by school level 
















Schools with achievement 


Other public schools 






scores 








Elementary 




19,794 


68.6% 


29,360 


56.6% 


Middle 




4,712 


16.3% 


8,179 


15.8% 


Secondary 




3,334 


11.5% 


12,377 


23.9% 


Combined 




1,047 


3.6% 


1,188 


2.3% 


Ungraded 




0 


0 


750 


1.4% 








100% 




100% 
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Table 6 compares the geographic distribution of schools included on the SASS 
student achievement subfile to that of SASS schools generally. (Note that the CCD 
categories used are defined based on 1984 census classifications. By 1993-94, some 4| 

school locales may have changed.) As can be seen in Table 6, rural schools were less 
likely than others to be included on the pilot SASS student achievement subfile (21 
percent, vs. 29.7 percent). Thus, analyses that ignore the locales of the schools will 
slightly tend to overrepresent urban schools. This difference is primarily due to the 
selection of states for the pilot study: the percentage of public schools that are rural in ^ 

those 19 states, based on CCD, is 20 percent, compared to 33 percent elsewhere. In 
completing the SASS 1993-94 student achievement subfile, an attempt will be made to 
add states with large rural populations. 



Table 6 — ^Weighted distribution of public schools with acbievement scores and other public schools, by 
locale 



Schools with achievement Other public schools 

^ores 



Large Central City 


3,675 


12.7% 


4,125 


8.0% 


Mid-size Central City 


4,594 


15.9% 


6,790 


13.1% 


Urban Fringe of Large City 


4,559 


15.8% 


7,174 


13.8% 


Urban Fringe of Mid-size City 


3,671 


12.7% 


4,240 


8.2% 


Large Town 


658 


2.3% 


1,611 


3.1% 


Small Town 


5,668 


19.6% 


12,508 


24.1% 


Rural 


6,062 


21.0% 


15,405 


29.7% 






100% 




100% 



SASS Reading Achievement Scale Definition 

Once the matching process was completed, one of several state assessment 
measures was selected for inclusion on the pilot SASS student achievement subfile. In 
each of the 19 states, a published school-level reading or reading-related state assessment 
score was identified. The assessment scores were based either on nationally standardized 
tests or on assessment instruments developed by the State Education Agency for use in the 
state. The measure selected was a total reading score, or in a few cases at upper grades, a 
language arts score; when there were several such scores, the one most highly correlated 
with the State NAEP reading score was selected. The standardization was carried out in 
three steps. 
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Step 1: Removal of state scale differences 

Because the test instruments were scaled differently in each state, use of the raw 
test scores for analyses aggregated across states would be meaningless. Schools in a state 
that scored its test from 500 to 1 500 would appear to be far more effective than schools in 
a state that scored its test from 0 to 100. To address this difference, the school mean 
scores were adjusted to have a mean of 0 (zero) and standard deviation of 1 (one), at each 
grade in each state (Equation 1.1). 

-z = (School Mean- State Mean^^)/ State Standard Deviation (1.1) 

The measure thus created, STATE-Z, would compare each school with other schools 
assessed at the same grade in the same state. The state mean and standard deviation were 
computed based on the SASS schools with assessment data in the state. This measure 
would be appropriate for analyses that pool within-state comparisons across states. 
However, this measure would not be sensitive to between-state relations between school 
characteristics and achievement. Moreover, to use this measure appropriately in cross-state 
analyses of SASS, one must also standardize the school characteristics to have the same 
mean and variance in each state — a nuisance, but not an insurmountable difficulty. 

Step 2: Incorporation of state differences based on NAEP 

To incorporate between-state variation in achievement into the SASS achievement 
measure. State NAEP was used. Starting with STATE-Z, the scores were multiplied by 
the standard deviation of NAEP school means in the state, and the state NAEP mean was 
added to all of the scores in that state (Equation 1.2). For this purpose, the 1994 State 
NAEP reading scale scores were used. The NAEP means and standard deviations were 
based on scores for schools that had been merged with state assessment data as a part of a 
previous project, though they could equally well have been based on the entire State 
NAEP sample in the state. To avoid confusion between the SASS student achievement 
measure and NAEP, the resulting school means were rescaled to a mean of 50 and a 
standard deviation of 20 (Equation 1.3). 

Y = {State -z)x (State NAEP Standard Deviation) + State NAEP Mean (1 .2) 

NATLPUB = (Y - Y) X (20 / Standard Deviation(Y)) + 50 (1 .3) 

The measure thus created, NATLPUB, would place each school on a common 
scale with other schools in the 19-state SASS sample. However, there would be no 
assurance that the reading achievement score measure in one state was measuring the same 
thing as the reading achievement measure in another state. Moreover, because the 4th 
grade State NAEP reading assessment was used for this step, extrapolations to 
achievement measures based on assessments of other grades are based on the assiunption 
that the ranking of states on reading achievement is similar across grade levels. If that 
assumption is faulty, then analyses of the relations of student achievement to school 



characteristics at other grades (e.g., in high schools) should not necessarily be expected to 
yield coherent results; while analyses within any one state would be meaningful, 
aggregating across states would be problematic. One set of analyses in this report 
addresses die reasonableness of this assumption (see the discussion of Research Question 
#5 in Chapter 3). 

Step 3: Projection of within-state variation 

As a final step, to align within-state variation more closely with the NAEP scale. 
Equations 1 .2 and 1 .3 were recomputed after the within-state deviation from the state 
mean was multiplied by the correlation between NAEP and the state assessment 
(Equations 1 .4 and 1 .5). State assessments can be expected to tap both domains covered 
by NAEP and other domains. The correlation between NAEP and state assessment scores 
provides a measure of the similarity between the tests. Thus, in a sense, multiplying the 
within-state deviation fi'om the state mean by the correlation between NAEP and the state 
assessment “projects” the state assessment variation onto the NAEP scale, capturing that 
part of the state assessment that was like NAEP. In a state in which the state assessment 
was poorly correlated with NAEP, variation in the SASS achievement measure woitid be 
reduced. One of the analyses presented in the report analyzes the correlations of state 
achievement scores with State NAEP in each state (see the discussion of Research 
Question #1 in Chapter 3). 



Y* = (State - z) X (State NAEP Standard Deviation x r(NAEP, State Assesment)) ^ 

+ State NAEP Mean 

ACHIEVE = (T* - ?* ) X (20 / Standard Deviation(T* )) + 50 (1 .5) 

The variable thus created, ACHIEVE, is theoretically appropriate for within- and 
across-state analyses of SASS. However, in states with assessments that are not highly 
correlated with State NAEP, the full variation in reading achievement between schools is 
not as likely to be represented as in states whose assessments are highly correlated with 
NAEP. Thus, in any analyses that examine correlations between SASS school 
characteristics and student achievement, data from the “highly correlated” states will be 
given somewhat greater weight de facto. 

It should be emphasized, in passing, that although State NAEP data were used to 
capture between-state variation in achievement, it woitid be highly misleading to interpret 
the SASS student achievement measure as a surrogate for a school’s average NAEP 
proficiency. Not only are the different state assessments differentially correlated with 
NAEP, but it must also be remembered that State NAEP assessments are administered 



One state, not included in the report, was deleted from consideration when the assessment scores that AIR 
requested from it were found to be correlated 0.18 with State NAEP. In that case, the scores were apparently 
for a mini mum competency measure, as school means ranged from 90% to 100%. 



with different procedures and a different set of student and school motivations from the 
procedures and motivations associated with state assessments. 



3 . 



SASS Student Achievement Analyses 



To evaluate the pilot SASS student achievement subfile and explore the value the 
addition of student achievement data would have for SASS, this chapter presents results 
of analyses designed to address five research questions: 

1 . How close are the NAEP-transformed scores to NAEP school means? 

2. How similar are SASS public schools with student achievement scores to 
other SASS public schools? 

3. Are there statistically meaningful school-level relations between the 
student achievement measure and SASS measures? 

4. Does incorporating between-state variation into the student achievement 
score, using NAEP data, increase the power of the student achievement 
measure? 

5. Can between-state variation in student achievement be extrapolated from 
elementary level scores to secondary level scores? 

The first question is concerned with the extent to which different state assessments 
correlate with a single overall measure of reading achievement, based on NAEP, while 
the second research question is concerned with the representativeness of the 2,575 SASS 
schools for which student achievement data were obtained. The final three questions 
focus on the utility and generality of the student achievement subfile. Can it be used for 



meaningful analyses relating school factors measured by SASS to student achievement? 
If so, are the student achievement measures improved by including between-state 
variation based on NAEP? Finally, can the between-state variation based on NAEP, 
which is derived from a 4th grade assessment sample, be generalized to secondary 
schools? 

Figure 1 presents a model summarizing potential links between average student 
achievement in a school and various factors addressed by SASS: student background 
characteristics, teacher qualifications, the nature of the school climate, and key 
organizational features of the school. Because the model will be helpful in responding to 
the research questions outlined above, we review it briefly before proceeding to a 
discussion of these questions. In this review, we summarize the support that can be found 
in the literature for an association between each of the factors represented in the model 
and student achievement and describe how the factors were operationalized in the study 
using available SASS measures. 



Figure 1 -Model relating SASS measures to achievement scores 
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Model and SASS Measures 



Student background characteristics 

Family backgroimd and socioeconomic status have consistently been shown to be 
related to student achievement (Coleman et al., 1966; Hanushek, 1986). To explain these 
relationships, researchers have called attention to differences in students’ health and 
nutrition, in the physical environments of their homes (e.g., noise levels), in the 
configurations of their families (e.g., single parent vs. dual parent households), in the 
parenting styles, beliefs, and expectations of their mothers and fathers, and in their 
inherited intelligence (for reviews, see Scott- Jones, 1984; Henderson and Berla, 1994). 

In studies of the relations between school attributes and achievement, student backgroimd 
characteristics are often included as control variables, capturing variation in academic 
performance that cannot be explained by differences in students’ school experiences. 
However, while largely outside of school influence, the impact these variables may have 
on achievement is not likely to be entirely independent of school arrangements. In many 
cases, students may be at risk of academic failure not because of their familial or 
socioeconomic backgrounds per se, but rather through a misfit between school offerings 
and demands and their particular needs and capabilities (Gordon and Yowell, 1994). 
Lareau (1987), for example, illustrated how the limited educational backgrounds and 
other resources of working-class parents interfered with their ability to meet school 
expectations regarding appropriate levels of involvement in their children’s schooling. 
Furthermore, as our model suggests, schools themselves may be influenced in multiple 
ways by the backgrounds of their students. Schools with students from more advantaged 
backgrounds may, for example, find it easier to recruit and retain highly qualified teachers 
and to maintain orderly and disciplined school environments. 

Measures. Several SASS measures were used to address student background 
characteristics, including the percentages of students in a school who are eligible for free 
lunch, receive Title I services, or are in remedial mathematics, remedial reading, special 
education programs, bilingual education programs, or English as a Second Language 
programs. In addition to these reported percentages, a community factor, capturing the 
family backgrounds of students in a school, was constructed by averaging teachers’ 
responses to three related attitudinal items concerned with the extent to which they 
perceive parental alcoholism and/or drug abuse, poverty, and racial tension to be 
problems in their schools.** Because a factor analysis found all of the student measures 
to be loaded on a single factor, a single student background composite was used in this 
study. The various SASS student background measures and the community factor were 
divided by their standard deviations and then averaged, as shown below: 



" A series of attitudinal items were included in the principal’s and teacher’s questionnaires for SASS. 
These items were factor analyzed to identify a set of factors that would provide more reliable information 
about teachers, administrators, and their schools than individual items. These factors were used in Private 
Schools in the United States: A Statistical Profile, 1993-94 (McLaughlin and Broughman, 1997). For a 
detailed description of the factors, see Appendix A. 



Student background composite = average of (100 - Free Limch Pet. / 29, 100 - 

Remedial Reading Pet. / 16, 100 - Remedial Math Pet. / 14, 100 - Bilingual 
Education Pet. / 11, 100 - ESL Pet. / 9, 100 — Special Education Pet. / 9, 100 
- Title I Pet. / 21, Community Factor / 1.73) 

Teacher qualifications 

To the extent that greater teacher experience, education, and training are 
associated with higher quality instruction, teacher qualifications might be expected to be 
related to student achievement. Indeed, recent studies have provided support for such a 
link. Turner and Camilli (1988), for instance, foimd teacher education and experience to 
be associated with students’ math and reading achievement. Monk (1994) similarly 
observed teachers’ subject matter preparation to be positively related to student learning 
gains. In keeping with these findings, we expect that teacher qualifications will be 
positively associated with student achievement. 

Measures. Because the state assessment data were merged with SASS at the 
school level, analyses must also be conducted at that level. The teacher qualification 
variables are, therefore, school-level averages computed from information provided by 
individual teachers. Available SASS qualification measvues include: (1) teachers’ years 
of experience, (2) teachers’ level of education (i.e., possession of a masters degree or 
not), (3) teachers’ participation in activities related to teaching (e.g., sponsored 
workshops or in-service progreuns, university extension or adult education covuses, 
college covuses in the subject field, professional growth activities sponsored by 
professional associations, committees to integrate academic skills into vocational 
curricula, or other curriculiun committees), (4) teachers’ participation in in-service or 
professional development programs, (5) teachers’ ratings of the impact professional 
development programs have had on them, and (6) teachers’ work load (i.e., the sum of the 
hovus teachers were required to be at school during their most recent full week of 
teaching and the hovus they spent after school, before school, and on the weekend in 
school-related activities, weighted by the standard deviations of the measvues). The SAT 
scores of inco min g freshmen at the colleges or universities attended by teachers were also 
obtained to capture the selectivity of these institutions.*^ From these available measvues, 
fovu teacher qualification factors were constructed for use in the present study: (1) 
Formal Qualifications (the weighted sum of years of teaching experience and level of 
education), (2) In-service Training (the weighted sum*^ of items addressing participation 
in activities related to teaching, participation in in-service or professional development 
programs, and impact of professional development), (3) Workload, and (4) Selectivity. 



The SAT scores of the teachers were obtained from another dataset, using infonnation from the SASS 
teacher questionnaire on the college attended. 

Each tenn in a sum of differently scaled tenns was divided by its standard deviation, to give variation on 
each term approximately equal weight in the sum. Thus, each term was effectively measured in standard 
deviation units. 



One might expect teachers’ salaries to be indicative of the quality of instruction 
students receive. However, schools with higher achieving students may find it easier to 
attract highly qualified teachers. The raw correlation between teacher salary and student 
achievement can, therefore, be misleading. Likewise, while one might expect that 
matching teachers to the subjects they feel most qualified to teach would lead to higher 
achievement, teachers’ reports of teaching in field can be influenced by the success of 
their students. Again, the raw correlation between reported teaching in field and student 
achievement can be misleading. 

Therefore, the qualification measures were combined into two separate 
composites: (1) the best predictors of salary variation, and (2) the best predictors of 
teachers’ reports of teaching in field. The first composite indicates the combination of 
qualifications that is most valued by school districts, and the second composite indicates 
the combination of qualifications possessed by teachers who are teaching what they 
consider themselves to be most qualified to teach. 

Teacher salary is the sum of teachers’ academic year base salary and any 
additional compensation received from the school for extracurricular or additional 
activities. The reported teaching in field measure assesses the correspondence between 
teachers’ main teaching assignments and the fields in which they feel most qualified to 
teach (reported teaching in field will equal 1 if the two SASS measures match and 0 if 
they do not). The predicted teacher salary and predicted teacher reports of teaching in 
field, linear combinations of the teacher qualification factors, were used to capture 
teacher qualifications in analyses. The specific linear combinations are shown below: 

Elementary: 

Teacher salary composite = (0.51 Formal Qualifications) + (0.06 In-Service 
Training) + (0.10 Work Load) + (0.30 Selectivity) 

Reported teaching in field composite = (-0.04 Formal Qualifications) + (-0.04 In- 
Service Training) + (-0.08 Work Load) + (0.02 Selectivity) 



Middle: 

Teacher salary composite = (0.48 Formal Qualifications) + (0.06 In-Service 
Training) + (0.04 Work Load) + (0.42 Selectivity) 

Reported teaching in field composite = (-0.04 Formal Qualifications) + (-0.02 In- 
Service Training) + (0.12 Work Load) + (0.06 Selectivity) 



Secondary: 



Teacher salary composite = (0.50 Formal Qualifications) + (0.10 In-Service 
Training) + (-0.08 Work Load) + (0.37 Selectivity) 



Reported teaching in field composite = (0.1 1 Formal Qualifications) + (-0.15 In- 
Service Training) + (0.13 Work Load) + (0.03 Selectivity) 



School climate 

One of the more consistent findings in the effective schools literature is that an 
orderly and disciplined school climate is necessary to provide students with adequate 
opportunities to learn. As Purkey and Smith (1983) conclude in their review of this 
literature, “common sense alone suggests that students cannot learn in an environment 
that is noisy, distracting, or unsafe” (p. 445). Boyer (1995) has similarly argued that 
maintaining the school as “a disciplined place” is essential for effective schooling in the 
early grades. “Children, like the rest of us,” he observes, “work best when they feel safe, 
when reasonable rules are sensitively established, thoughtfully explained, and reasonably 
enforced” (p. 25). An orderly school climate may not only enhance the performance of 
students but also the performance of teachers. Newmann, Rutter, and Smith (1989) have 
shown, for instance, that where student behavior is orderly teachers’ sense of efficacy is 
higher, as are the expectations they hold for their students. 

Measures. The school climate measure used in the present study incorporates six 
factors formed from principals’ ratings of the seriousness of various school problems on 
the SASS Principal Questionnaire (i.e., Basic Standards, such as drugs and dropping out. 
Respect for Teachers, Community, Attendance, Apathy, and Respect for Property) and 
five factors formed from teachers’ ratings of the seriousness of various school problems 
on the SASS Teacher Questionnaire*'* (i.e., Basic Standards, Respect for Teachers, 
Attendance, Respect for Students, Behavior Standards). These factors were averaged 
together wdth an additional factor formed from the appropriate average of three other 
SASS items (teachers’ reports of tardiness in their classes, whether they have ever been 
threatened by students, and whether they have ever been attacked by students), weighting 
by factor standard deviations as shown below: 

School climate measure = average of (Principal’s perception of school problems / 
1.61, Teachers’ perception of school problems / 0.75, Teachers’ perception 
of student behavior / 0.14) 

School organization 

Our model calls attention to relationships between student achievement and three 
aspects of school organization: (1) school size, (2) class size, and (3) principal and 
teacher control over the operation of the school (as reported by the principal), along wdth 
teacher influence on school policies and control of classroom practices (as reported by 
teachers). 
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As Lee et al. (1993) note, two conflicting perspectives can be found in the 
literature on school size, one highlighting positive consequences of increased size (e.g., 
economies of scale and greater diversity of program offerings) and the other hi ghl ighting 
negative consequences (e.g., more formalized and impersonal social interactions). Recent 
empirical research is, however, more supportive of the latter perspective than the former. 
Larger grade sizes have been shown to be associated with reduced academic engagement 
among eighth-grade students (Lee and Smith, 1993). Similarly, at the high-school level 
school size has been found to be negatively related to students’ early achievement gains 
(Lee and Smith, 1995), along with students’ participation in school activities, satisfaction 
with school experiences, and school attendance (Lindsay, 1982). Dropout rates have also 
been shown to be higher in large high schools than in small high schools (Pittman and 
Haughwout, 1987). To provide students with a more stable and supportive educational 
experience, many reformers have called for dividing large schools into smaller subunits 
(e.g., Oxley, 1994). Likewise, Boyer (1995) has argued that to be effective, elementary 
schools must “be small enough for everyone to be known by name — ^perhaps with no 
more than four to five hundred students” (p. 1 7). Guided by recent research, we expect 
that school size will be negatively related to student achievement. 

We similarly anticipate that class size will be negatively associated with student 
achievement. Meta-analyses of existing studies (Glass et al., 1982), along with large- 
scale research projects based on experimental designs (Firm and Achilles, 1990), have 
confirmed a link between class size and student learning (Mosteller, Light, and Sachs, 
1996; for a critical review, see Slavin, 1989). Blatchford and Mortimore (1994) offer a 
number of possible explanations for the relationship between class size and student 
achievement. Small class size may facilitate more individualized and effective 
instruction, more complete curriculum coverage, and greater student involvement in 
classroom activities. 

The effective schools literature calls attention to the importance of school-site 
management for student achievement (Purkey and Smith, 1983). To respond adequately 
to the needs of their students, it is argued, principals and teachers must be able to make 
many educational decisions for the school independent of district- or state-level officials. 
This literature also highlights the importance of instructional leadership at the school site 
for school effectiveness (Purkey and Smith, 1983), along with the ability of teachers to 
respond in innovative ways to instructional problems and challenges (Wehlage et al., 

1 989). It is, therefore, of interest to explore whether student achievement is higher where 
principals and teachers report having greater control over the operation of the school. 

Measures. School size was measured by the total number of students enrolled in 
the school. A class size factor was constructed by averaging teachers’ reports of then- 
class sizes, teachers’ reports of their satisfaction with their class sizes, and school-level 
measures of student/teacher ratios (weighted in terms of the standard deviations of the 
measures). Class size measures for schools were generated by averaging the reports of 
individual teachers. For a teacher teaching in a single self-contained classroom, class size 
is the total number of students emolled in the teacher’s class at the school. For a teacher 



teaching in multiple departmentalized classrooms, by contrast, class size is the average 
number of students in the teacher’s classes. The student/teacher ratio is calculated as the 
total number of students in a school, divided by the sum of the number of full-time 
teachers in the school and one-half the number of part-time teachers in the school. 

Finally, teacher’s satisfaction with class size is captured by an item in the SASS Teacher 
Questionnaire asking teachers whether they are satisfied with the size of their classes; 
responses ranged from 1 (“Strongly Agree”) to 4 (“Strongly Disagree”). 

Principals’ perceptions of the control they and teachers have over the operation of 
their schools were captured through items on the SASS Principal Questionnaire asking 
principals how much actual influence they think each of several groups (e.g., the State 
Department of Education, district staff members, the school board, library media 
specialists/librarians, parent associations, the principal, and teachers) has on decisions 
related to establishing curriculum, hiring new full-time teachers, setting discipline policy, 
deciding how the school budget will be spent, determining the content of in-service 
programs, and evaluating teachers. The measures are scaled from 0 (“None”) to 5 (“A 
Great Deal”). Principals’ ratings of their own control over these decisions and their 
ratings of teachers’ control over these decisions were averaged separately (dividing each 
item’s score by its standard deviation) to form two control measures. 

A separate measure of teachers’ control in the classroom includes two factors 
formed from attitudinal items on the SASS Teacher Questionnaire: teachers’ influence 
on curriculum (i.e., in establishing curriculum, in selecting textbooks and other 
instructional materials, and in selecting content, topics, and skills to be taught) and 
teachers’ classroom control (i.e., in selecting teaching techniques, in evaluating and 
grading students, in disciplining students, and in determining the amount of homework to 
be assigned). These factors were appropriately averaged together with an additional item 
asking individuals whether they would choose to become teachers once more if they were 
begiiming their careers all over again. 



Results of Preliminary Analyses 

We now turn to a discussion of the five research questions addressed in the pilot 

study. 



1. How close are the NAEP-transformed scores to NAEP school means? 

Within states, the information in the SASS student achievement measure is ^ 

provided by state assessments; and between states, the information in the SASS student 
achievement measure is based on information provided by NAEP. Although all state 
assessments included in the file are English reading/language arts scores, they may differ 
substantially in format and content. However, to the extent that they measure s imil ar 
constructs of student achievement, the results of analyses using the measure can be • 

aggregated across states to represent a large portion of the population of American public 
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schools. Prior to the construction of the SASS student achievement measure, the (within 
state) correlations between NAEP and state assessment scores were computed based on 
9' the entire NAEP 4th grade sample in each state, and, as shown in Table 7, these 

correlations ranged between .32 and .86, with a median value of .66. For one state, not 
included in the SASS student achievement subfile, the correlation was .18. 



Table 7 — Correlations between state assessment and NAEP school means in reading in 1993-94, by 
state 






State 


Number in 
Overlap Sample 


Correlation, based on 
Overlap Sample 


Correlation, based on Entire 
NAEP Sample 


Alabama 


17 


.67 


.72 


California 


3 


.90 


.78 


Delaware 


14 


.14 


.32 


Florida 


4 


.99 


.85 


Georgia 


6 


.63 


.66 


Hawaii 


39 


.55 


.65 


Kentucky 


10 


.81 


.37 


Louisiana 


14 


.83 


.84 


Massachusetts 


8 


.94 


.82 


Maine 


18 


.57 


.63 


Michigan 


2 


— 


.73 


Montana 


27 


.19 


.50 


New York 


0 


— 


.39 


Rhode Island 


27 


.91 


.86 


Tennessee 


6 


.49 


.63 


Texas 


8 


.60 


.72 


Washington 


5 


.91 


.73 


Median 




.67 


.66 



SOURCE: 1994 NAEP Trial State Assessment in reading; state assessment files for 17 states. 



To assess the extent to which the measures are similar for schools in the SASS 
sample, the correlations between the student achievement measure and the NAEP sample 
measure were also computed based on the overlap sample of 208 schools in 1 5 states. 
Because the overlap sample is a subsample of the entire (typically 100-school) NAEP 
sample in each state, the two sets of correlation coefficients (shown in Table 7) are 
somewhat different. The overall correlation based on the SASS/NAEP overlap sample 
was .73. To a certain extent, this correlation is a tautology, because the NAEP state 
means (for the sample) were used in the construction of the SASS student achievement 
measure. However, the median within-state correlation was also a fairly strong .67, 
indicating that, on average, a typical state assessment “sorts” schools similarly to NAEP. 
This correlation based on the overlap sample was comparable to the median within-state 
correlation based on the entire NAEP sample in each state, .66. 

There was noticeable variation in correlations between states. Based on the total 
NAEP samples, correlations in thirteen states ranged from .63 to .86; however, in four 
other states, the correlations ranged from .32 to .50. Correlations based on the much 
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smaller overlap sample were more variable. In seven states, the correlations based on the 
overlap sample ranged from .81 to .99; in six of the states, the correlations ranged from 
.49 to .67; however, in two states, the correlations were only .14 and .19. Thus, one 
should not expect analytical results to be uniform across states. Nevertheless, in 
aggregate, when one recognizes that the NAEP school means have built-in error due to 
sampling and that both NAEP and state assessments have measurement error, these 
correlations must be considered acceptable as a basis for use of the SASS student 
achievement measure in research. 

Within the overlap sample of 208 schools, the SASS student achievement school 
means typically differed from (a multiple of)*^ the NAEP school mean by about 6.3 
points, a difference more than three-quarters as large as the standard deviation of the 
SASS student achievement school means for this sample (7.8 points). On the NAEP 
proficiency scale, such a difference would be 12.7 points. Thus, one would not be 
justified in considering the SASS school-level achievement measure to be a reliable 
estimate of an individual school’s performance on NAEP. Nevertheless because all of the 
reading achievement scores are correlated with NAEP, the SASS student achievement 
measure appears to have substantial commimality across states. 

2. How similar are SASS public schools with student achievement scores to 
other SASS public schools? 

How similar are SASS public schools for which we have student achievement 
scores to other SASS public schools? If the SASS public schools with student 
achievement scores are substantially different fix)m other SASS public schools, then the 
generalizability of the results of analyses of the SASS student achievement subfile may be 
limited, and achievement data from a larger munber of states may be needed to increase 
the representativeness of the data set. 

Differences observed might be due to differences between schools in the 19 states 
represented in the study and schools in other states or to differences between schools with 
achievement scores in these 19 states and other SASS schools in the same states. 
Therefore, we first compare all 3,668 SASS public schools in the 19 states (both with and 
without achievement scores) with the 5,099 SASS public schools in the other states. 

Then, to test for within-state differences, the 2,575 SASS public schools with 
achievement scores in the 19 states are compared with the 1,093 SASS public schools 
without achievement scores in these same states. Results of the t-tests are shown in 
Tables 8a and 8b. (Note: separate t-tests were also carried out within elementary, middle, 
secondary, and combined school groupings, and the results were similar). 



The SASS student achievement measure was scaled to a different range from NAEP, as described in 
Chapter 2. For this comparison, NAEP was scaled to match the SASS student achievement scale. 



Student Characteristics 



Cross-State Comparison The information displayed in Table 8a indicates that 
there are statistically significant differences’^ on thirteen of the fourteen student 
characteristic measures between the SASS public schools in the 19 states represented in 
the study and SASS public schools in other states. The SASS schools in the 19 states 
have higher percentages of students from minority ethnic groups, students in English as a 
second language and bilingual education programs, students in free limch and Chapter 1 
programs, students in remedial math and reading programs, and students from 
disadvantaged community backgroimds. SASS schools in the 19 states have fewer white 
students and fewer American Indian/Alaskan Native students. The mean percentage 
differences between SASS schools in the 19 states that have student achievement scores 
and other SASS public schools range from 1 percent to 5 percent, except for Hispanic 
eiuollment, where the difference is 9 percent. One plausible explanation for some of 
these differences in student characteristics is that the 19 states represented in the study 
include California, New York, and Texas, all states that enroll many students from 
minority ethnic groups. Based on CCD, the average percentage of Hispanic students in 
the 19 states included on the subfile was 17 percent, compared to 6 percent in the states 
not included. By contrast, the average percentage of African American students was 16 
percent on both subfiles. 

Within-State Comparison. SASS schools with achievement scores differ 
significantly from other SASS public schools in the same 19 states on only six SASS 
student characteristic measures. The mean percentage differences range from 1 percent to 
5 percent for most cases. SASS schools with achievement scores have higher percentages 
of Asian/Pacific Islander students and students in LEP, ESL, and free limch programs, 
and fewer white students and students in programs for students with disabilities, than 
other schools in the same states. Some of these differences may reflect the fact that many 
of the SASS schools that could not be matched with state assessment data are special 
education schools. 

Teacher Characteristics 

Cross-State Comparison. Teachers’ salaries and adjusted salaries are higher in 
the 19 states represented in the study than in other states (note that the SASS achievement 
subfile includes New York, Hawaii, and California, which have the highest cost of living 
of all states, so the differences are smaller for adjusted salaries). In addition, teachers in 
the 19 states participate more in professional development, feel that they receive less 
support for professional development, participate less in activities related to teaching, feel 
less qualified for their teaching assignments, and have fewer weekly hours of student 
interaction than teachers in other states. Schools in the 19 states also have more difficulty 
filling vacancies than schools in the other states. 



The Bonferroni correction was applied to adjust for the increased likelihood of erroneously finding 
differences significant resulting from the large number (61) of comparisons. For a=0.05, the Bonferroni- 
adjusted critical t value is 3.18. 



Within-State Comparison. Teacher salary and adjusted salary are the only 
variables that show a difference between SASS schools with achievement data in the 19 
states and other SASS schools in the same states. The lower pay teachers receive in the 
SASS schools for which we obtained state assessment data may be explained by the 
overrepresentation of elementary schools among assessed schools (See Table 5). 



Table 8a — Comparison between SASS public schools in 19 states with state assessment scores and 
other SASS public schools 





Mean SASS measures 
for the 19 states 


Mean SASS measures 
for the other states 


19 states in the study 
vs. other states 


Student Characteristics 
Hispanic, regardless of race 


0.14 


0.05 


14.6237* 


Black, not of Hispanic origin 


0.15 


0.14 


3.3998* 


Asian/Pacific Islander 


0.04 


0.01 


12.2898* 


White, not of Hispanic origin 


0.66 


0.77 


-14.7695* 


American Indian/Alaska Native 


0.01 


0.02 


-4.4863* 


L£P students 


0.06 


0.02 


10.3152* 


English as a second language 


0.05 


0.02 


8.7932* 


Bilingual education 


0.04 


0.02 


3.4719* 


Free lunch program 


0.42 


0.37 


5.1178* 


Chapter 1 


0.18 


0.13 


6.1146* 


Remedial reading 


0.14 


0.12 


4.0033* 


Remedial math 


0.09 


0.07 


4.0848* 


Programs for students w/disabilities 


0.08 


0.08 


-0.3243 


Conununity background 


5.57 


5.87 


-5.7380* 


Teacher Characteristics 
Mean teacher college SAT scores 


925.91 


914.79 


0.0592 


Years of teaching experience 


14.79 


14.80 


-0.0836 


Adjusted teacher salary 


34,397.45 


32,323.28 


11.5172* 


Teacher salary 


34,582.11 


31,170.88 


19.7179* 


Teaching in field feel best qualified 


0.77 


0.81 


-6.0354* 


Weekly hours-other school activity 


8.51 


8.45 


0.5200 


Hours at school per week 


31.61 


31.84 


-1.1132 


Weekly hours student interaction 


2.75 


3.15 


-4.8807* 


Participated in activities related to 
teaching 


3.47 


3.65 


-5.3370* 


Opinion about impact of professional 
development ^ 


13.31 


13.32 


-0.0029 


Participation in professional 
development 


3.92 


3.50 


7.7278* 


Support received for professional 
development 


1.97 


2.09 


-5.5616* 


Difficult filling vacancies ^ 


6.93 


6.09 


4.6217* 



Note: * indicates that the differences are statistically significant 
Note: ^ higher numbers indicate more negative situations 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 1993-94 Schools and Staffing 
Survey (Public School Teacher Questionnaire, Principal Questionnaire, and School Questionnaire). 



Table 8a — Comparison between SASS public schools in 19 states with state assessment scores and 
other SASS public schools, continued 





Mean SASS measures 


Mean SASS measures 


19 states in the study vs. 




for the 19 states 


for the 33 states 


other 33 states 


School Climate 
Teacher: basic standards 


7.73 


7.68 


1.3077 


Teacher: attendance 


6.71 


7.00 


-6.9169* 


Teacher: behavior standards 


6.88 


6.73 


0.1085 


Teacher: respect for students 


5.94 


5.99 


-0.8105 


Teacher: satisfaction 


6.98 


7.07 


-1.6463 


Teacher: respect for teachers 


6.43 


6.58 


-3.2068* 


Attacked by student last year 


1.95 


1.96 


-1.1223 


Ever physically attacked by student 


1.89 


1.90 


-1.1013 


Ever threatened by students 


1.77 


1.78 


-0.8762 


Number of students tardy per wk. 


4.18 


3.24 


6.4066* 


Class interruptions per week 


17.15 


15.04 


0.6374 


Principal: basic standards 


8.57 


8.49 


2.3423 


Principal: community 


6.32 


6.56 


-3.9323* 


Principal: attendance 


6.74 


7.12 


-6.3125* 


Principal: apathy 


6.42 


6.53 


-1.8310 


Principal: respect for teachers 


7.84 


7.94 


-0.0622 


Principal: respect for property 


7.83 


8.11 


-5.5988* 


Student non-attrition rate 


1.11 


1.11 


-0.0520 


School Organization 
Student/teacher ratio 


17.92 


16.19 


0.4971 


Average class size 


23.64 


21.67 


10.1659* 


Total enrollment 


585.13 


448.75 


15.6290* 


Teacher satisfied with class size ^ 


2.22 


2.06 


7.7838* 


Teachers: influence on budget 


3.19 


2.96 


6.2928* 


Teachers: determine in-service 
program 


3.90 


3.93 


-1.0269 


Teachers: influence discipline 
policy 


4.01 


4.01 


-0.1972 


Teachers: influence on hiring 


2.68 


2.39 


6.3050* 


Teachers: influence evaluating 
teachers 


2.11 


2.28 


-3.8787* 


Teachers’ influence on curriculum 


3.58 


3.72 


-3.7910* 


Teachers’ classroom control 


6.49 


6.52 


-1.3043 


Principal: determine in service 
program 


3.95 


3.91 


0.0457 


Principal: influence on hiring 


4.29 


4.32 


-1.0895 


Principal: influence on discipline 
policy 


4.35 


4.38 


-1.1981 


Principal: influence on budget 


3.81 


3.60 


6.1744* 


Principal: influence evaluating 
teachers 


4.69 


4.73 


-1.7109 


Principal: influence on curriculum 


3.43 


3.45 


-0.5236 


Sample size 


3,668 


5,099 


3,668 vs. 5,099 



Table 8b — Comparison between SASS public schools with student achievement scores and other 
SASS public schools in the same state 





Mean with 
achievement scores 


Mean without 
achievement scores 


With achievement 
scores in 19 states vs. 
without achievement 
scores 


Student Characteristics 
Hispanic, regardless of race 


0.15 


0.11 


3.02705 


Black, not of Hispanic origin 


0.16 


0.14 


1.01288 


Asian/Pacific Islander 


0.04 


0.02 


5.24921* 


White, not of Hispanic origin 


0.64 


0.71 


-3.95653* 


American Indian/Alaska Native 


0.01 


0.01 


-1.51437 


LEP students 


0.07 


0.04 


3.85667* 


English as a second language 


0.05 


0.03 


3.34799* 


Bilingual education 


0.04 


0.02 


2.14531 


Free lunch program 


0.44 


0.36 


4.90641* 


Chapter 1 


0.19 


0.15 


3.16017 


Remedial reading 


0.14 


0.13 


1.12654 


Remedial math 


0.09 


0.08 


0.94314 


Programs for students w/disabilities 


0.07 


0.11 


-5.11452* 


Community background 


5.56 


5.62 


-0.62165 


Teacher Characteristics 
Mean teacher college SAT scores 


918.5 


946.06 


-0.14477 


Years of teaching experience 


14.58 


15.35 


-2.37028 


Adjusted teacher salary 


33,557.08 


36,680.62 


-7.59162* 


Teacher salary 


33,456.99 


37,638.89 


-9.28059* 


Teaching in field feel best qualified 


0.78 


0.75 


2.27736 


Weekly hours-other school activity 


8.54 


8.42 


0.44595 


Hours at school per week 


31.73 


31.29 


1.21824 


Weekly hours student interaction 


2.69 


2.92 


-1.58161 


Participated in activities related to 
teaching 


3.48 


3.44 


0.60555 


Opinion about impact of professional 
development ^ 


13.28 


13.40 


-0.04419 


Participation in professional 
development 


4.00 


3.72 


2.53940 


Support received for professional 
development 


1.99 


1.92 


1.47531 


Difficult filling vacancies ^ 


6.97 


6.82 


0.40096 



Note: * indicates that the differences are statistically significant 
Note: ^ higher numbers indicate more negative situations 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 1993-94 Schools and Staffing 
Survey (Public School Teacher Questionnaire, Principal Questionnaire, and School Questionnaire). 



Table 8b— Comparison between SASS public schools with student achievement scores and other 
SASS public schools in the same state, continued 





Mean with 


Mean without 


With achievement 
scores in 19 states vs. 
without achievement 




achievement scores 


achievement scores 


scores 


School Climate 
Teacher: basic standards 


7.94 


7.14 


7.30338* 


Teacher: attendance 


6.76 


6.58 


1.79842 


Teacher: behavior standards 


6.91 


6.79 


0.08703 


Teacher: respect for students 


5.90 


6.04 


-1.05050 


Teacher: satisfaction 


6.97 


7.02 


-0.47318 


Teacher: respect for teachers 


6.44 


6.41 


0.40292 


Attacked by student last year 


1.96 


1.93 


2.37096 


Ever physically attacked by student 


1.90 


1.87 


2.31197 


Ever threatened by students 


1.79 


1.72 


3.77551* 


Number of students tardy per wk. 


3.78 


5.27 


-3.58306* 


Class interruptions per week 


17.73 


15.58 


0.60344 


Principal: basic standard 


8.81 


7.91 


9.78534* 


Principal: community 


6.30 


6.36 


-0.56166 


Principal: attendance 


6.78 


6.64 


1.28293 


Principal: apathy 


6.45 


6.33 


0.95423 


Principal: respect for teachers 


7.85 


7.80 


0.03095 


Principal: respect for property 


7.83 


7.81 


0.18126 


Student non-attrition rate 


1.11 


1.10 


0.89841 


School Organization 
Student/teacher ratio 


18.33 


16.81 


0.42503 


Average class size 


23.97 


22.74 


2.39036 


Total enrollment 


587.31 


579.21 


0.45278 


Teacher satisfied with class size ^ 


2.26 


2.12 


3.42154* 


Teachers: influence on budget 


3.20 


3.16 


0.55305 


Teachers: determine in-service 
program 


3.88 


3.96 


-1.33436 


Teachers: influence discipline 
policy 


3.99 


4.05 


-1.26478 


Teachers: influence on hiring 


2.65 


2.77 


-1.39270 


Teachers: influence evaluating 
teachers 


2.12 


2.06 


0.70914 


Teachers" influence on curriculum 


3.50 


3.81 


-5.02006* 


Teachers’ classroom control 


6.46 


6.56 


-2.10792 


Principal: determine in service 
program 


3.95 


3.95 


-0.00559 


Principal: influence on hiring 


4.28 


4.32 


-0.86054 


Principal: influence on discipline 
policy 


4.34 


4.39 


-1.19416 


Principal: influence on budget 


3.80 


3.83 


-0.66493 


Principal: influence evaluating 
teachers 


4.67 


4.76 


-2.78376 


Principal: influence on 
curriculum 


3.35 


3.65 


-5.16164* 


Sample size 


2,575 


1,093 


2,575 vs. 1,093 



School Climate 



Cross-State Comparison. Principals in the 1 9 states represented in the study 
reported a larger number of students tardy per week and more problems with tardiness, 
poverty, and respect for property than principals in other states. Teachers in the 19 states 
observe more problems related to student attendance and report that students show less 
respect for teachers. These differences may, in part, be explained by the fact that states 
represented in the study contain somewhat more central city schools than do the other 
states (see Table 6). In general, public school teachers and principals who work in central 
city schools were more likely than their counterparts in other types of communities to 
report problems of these types. 

Within-State Comparison. Differences between the SASS schools with 
achievement scores in the 1 9 states and other SASS schools in the same states are evident 
on only four of the twenty school climate measures. In schools with achievement scores, 
teachers and principals observe fewer problems with basic standards (e.g., drugs, alcohol 
dropping out, pregnancy), teachers report feeling threatened by students less often, and 
there is less student tardiness. 

School Organization 

Cross-State Comparison. Significant cross-state differences occur on measures of 
school size, average class size, teachers’ satisfaction with class sizes, and teachers’ and 
principals’ influence over school policies. SASS schools in the 19 states represented in 
the study have greater total enrollments and larger class sizes. Teachers in these states are 
less satisfied with the sizes of their classes than are teachers in other states. In addition, 
they report less influence over curriculum and the evaluation of other teachers and more 
influence over hiring decisions. Principals and teachers in these states also report having 
more control over school budgets than their colleagues in other states. 

Within-State Comparison. Teachers in the SASS schools with achievement scores 
are less satisfied with the sizes of their classes than are teachers in other schools in the 
same states. Principals and teachers in schools with achievement scores both indicated 
that they have less influence on curriculum. 

Conclusion 

The SASS achievement schools in the 19 states are somewhat different fi^om the 
remaining SASS public schools, especially in student inputs and school organization. A 
larger percentage have many minority, ESL, bilingual education students, and students 
with less advantaged backgrounds. They tend to be in more central areas with larger class 
size and less teacher satisfaction on class size. The addition of a few states, especially 
those states which have more schools in rural areas, will add credibility to generalizations 
based on the research. 



The difference between SASS schools with achievement scores in the 19 states 
and other SASS public schools in the same states are not remarkable. On only 16 out of 
62 SASS measures are there statistically significant differences. SASS schools with 
achievement scores tend to be elementary schools and to have more Asian/pacific 
Islander, LEP, ESL bilingual education, and Chapter 1 students, but fewer students in 
programs with disabilities. Teacher are paid significantly less, and teachers are less 
satisfied with class size. 

Fewer SASS-measure comparisons are significantly different within states than 
across states. One plausible explanation is the larger sample size in the across-state 
comparison (3,668 vs. 5,099) than the within-state comparison (2,575 vs. 1,093). The 
mean differences, also shown in Tables 8a and 8b, are similar sizes for both sets of 
comparisons. 

3. Are there statistically meaningful school-level relationships between SASS 
measures and student achievement? 

A test of the utility of the SASS student achievement measure is whether its use in 
analyses involving other SASS measures yields meaningful results. To address this 
question, we present some preliminary analytical results. 

Consider two possible perspectives on education. One perspective, captured by 
the model presented in Figure 1 , suggests that student background characteristics and 
school attributes both contribute meaningfully to student achievement. Students’ 
backgrounds affect their achievement by affecting their readiness to learn and their in- 
school behavior. Teachers’ qualifications and the climate and organization of the school 
similarly influence student learning by determining the quality of the instruction students’ 
receive and the quality of their school experiences. Student achievement is an outcome 
determined by all of the other factors. 

A second perspective, by contrast, implies that achievement is almost exclusively 
a function of students’ background and that aspects of the school are a function of the 
surrounding community and the value placed within it on education. Where home and 
neighborhood environments are fi-ee of problems and provide strong academic supports, 
student achievement is high; where these environments are less conducive to learning, 
achievement can be expected to suffer. Schools, according to this perspective, do not 
influence student achievement so much as they are influenced by it. Higher achieving 
students from more advantaged backgrounds behave better in school which, in turn, leads 
to more orderly and disciplined school environments, easier recruitment of highly 
qualified teachers, and greater principal and teacher control over school operations. 
Student achievement is an indicator of community values, and school characteristics are 
determined by those values. 

Both of these competing theoretical perspectives provide explanations for 
observed relations between student, teacher, and school characteristics and student 



achievement. Despite the impossibility of drawing clear causal inferences that decide 
between these perspectives from cross-sectional correlational data, one can still gain 
substantial information from the analyses. The perspectives can serve to frame discussion 
of results, highlight possible alternative explanations for various findings, and point the 
way for controlled research projects. 

Figxires 2.1, 2.2, and 2.3 present path coefficients associated with the model 
relating SASS variables to student achievement, separately for elementary, middle, and 
high schools. The likelihood of Type I errors in evaluating the significance of individual 
relationships can be expected to rise with the number of models considered. Accordingly, 
in examining results oxir focus is on patterns of relationships evident across the three 
school levels. Below, we consider findings for each of the foxir classes of variables 
represented in oxir model in turn. The indications of statistical significance in these 
preliminary analyses are based on average estimated design effects. In follow-up 
analyses, we will use standard SASS error estimation procedxires (balanced repeated 
replications) to assess the significance of regression coefficients. 
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Findings 



Student background characteristics. At all three school levels, student 
background characteristics display strong relationships to student achievement. Notably, 
however, these relationships appear to weaken somewhat from elementary school to 
middle school to high school. At least three explanations for the latter fmding are 
possible: (1) achievement may be more highly linked to students’ backgrounds in the 
early grades because the subject matter is more likely to be encountered and mastered 
outside of school; (2) the stronger association in the early grades may reflect the more 
limited time these students have been exposed to school influences, implying that over 
time schools have a leveling effect on individual differences; or (3) it may simply be the 
case that achievement tests used in the early grades are more sensitive to student 
differences than achievement tests used in the upper grades. 

School climate. As predicted, more orderly and disciplined school climates are 
associated with higher levels of achievement in elementary, middle, and high schools. In 
contrast to the results for student background characteristics, however, relationships of 
climate to achievement appear to strengthen with school level. Once again, multiple 
explanations for this pattern of findings are possible: (1) disciplinary problems may have 
a greater effect on achievement in the higher grades (the greater independence of students 
and potential severity of problems may make maintaining an orderly school environment 
more important in the higher grades); (2) disciplinary problems may have cumulative 
effects on students’ achievement over time; or (3) perhaps, in fact, poor achievement 
leads to poor behavior, and the negative effects of poor achievement on behavior are 
more pronounced in the upper grades (as might be the case if poor performance in school 
has a greater downward influence on students’ educational expectations and aspirations in 
the upper grades). 

Teacher qualifications. Measures of teachers’ qualifications are positively related 
to student achievement at all three school levels, with the exception of the salary- 
predictive composite, which is negatively related to achievement in high schools. The 
latter result is both unexpected and difficult to explain. The negative association between 
the salary-predictive composite and achievement at the high school level is observed 
whether the model is run with weighted or unweighted data and does not appear to be 
attributable to collinearity between the two teacher qualification measures; the 
correlations between the salary-predictive composite and the reported-teaching-in-field- 
predictive composite at all three levels are modest, with absolute values less than 0.25. 
The relationships between these measures and achievement require further study using 
SASS. Also, it should be added, the raw correlation between average teacher salary and 
average student achievement is negative in the high school sample. One very positive 
interpretation of this finding would be that equity issues are attracting the most highly 
qualified teachers to the high schools where they are needed most; that is, to the schools 
where student achievement has been lowest. However, this is but one of several possible 
competing inferences. 



School organization. Class size is inversely related to student achievement in 
elementary, middle, and high schools. However, while most extant research suggests that 
class-size reductions yield the greatest achievement gains in the early grades, our results 
suggest instead that the relationship between class size and achievement strengthens with 
school level. While somewhat surprising, this pattern is perhaps understandable in the 
context of our findings regarding school climate. The relationships of both climate and 
class size to achievement appear to grow in magnitude from elementary to middle to high 
school, suggesting that school factors may, in general, be more importantly related to 
achievement in the later grades than in the early grades. Alternatively, it may be that 
affluent communities, with high achieving students, place a greater premiiun on small 
class sizes in the later grades than do other communities. 

School size relates differently to achievement at different school levels. While 
school size is positively associated with achievement at the high school level, it is 
negatively associated with achievement at the middle school level (though not quite 
significantly). Among elementary schools, school size and achievement were similarly 
found to be negatively related when the model was run with unweighted data. With 
weighted data, however, no significant relationship is evident at the elementary level, for 
reasons yet to be determined. Taken together, these results may suggest that while 
elementary and middle school students benefit from the more personalized educational 
experiences associated with smaller schools, at the high school level the greater diversity 
of offerings and opportunities associated with larger schools becomes more important. 
Alternatively, achievement may be greater in urban settings, where large high schools 
tend to be found, than in rural settings. 

The three variables concerned with principal and teacher control over the 
operation of the school were not found to be consistently related to achievement across 
the three school levels. While it may be that these variables are not associated with 
student achievement, it is also possible that they have other, indirect effects not 
represented in our model or that controlling for factors, such as the characteristics of 
principals and teachers, would reveal their associations with achievement. 

Comments 

The effective schools literature has been criticized for relying heavily on case- 
study data and small samples of schools thought to be functioning extremely well or 
extremely poorly (Purkey and Smith, 1983). The SASS student achievement subfile, by 
contrast, provides an opportunity to explore factors that relate to student achievement 
with a much more representative base of data on several thousand public schools. 

Though unambiguous causal inferences cannot be made, student background 
characteristics and school attributes are both found to relate in important ways to 
achievement. 

The effective schools literature has also been criticized for overlooking the fact 
that school experiences may have very different outcomes for different student 



populations (Purkey and Smith, 1983). The SASS student achievement subfile can be 
used to consider how school environment, teacher qualifications, school organization, or 
other aspects of schooling measurable with SASS variables affect different types of 
student bodies. One avenue for future work is to explore the extent to which the 
relationships of these variables to achievement are moderated by the characteristics of the 
student bodies of schools. 

4. Does incorporating between-state variation into the student achievement 
score, using NAEP data, increase the power of the student achievement measure? 

As described in Chapter 2, there are two achievement measures: a within-state 
measure (“State-z) and a between-state measure (“Achieve”). Achievement scores from 
state assessments, along with all SASS measures, were first standardized by state and 
grade level for within-state school-level analyses. State school-level variation was then 
incorporated by adding the NAEP state mean to scores for schools in each state and 
multiplying the within-state deviations by (a) the NAEP standard deviation in the state 
and (b) the correlation between NAEP and the state assessment in the state. The results 
are shown in Table 9. Omitting the correlation factor, one obtains the results shown in 
Table 10. Omitting the NAEP-adjustment completely, one obtains the results shown in 
Table 11. 

The incorporation of between-state variation using NAEP increases the power of 
the student achievement measure. For all school types except middle schools, the 
adjusted associated with models relating SASS measures to student achievement are 
higher where the NAEP-adjusted achievement scores are used rather than the 
standardized scores. Comparing Tables 9 and 1 1, the SASS measures explain 5%, 6%, 
and 6% more of the NAEP-adjusted achievement scores than the within-state 
standardized scores for elementary, secondary, and combined schools, respectively. (For 
middle schools, the difference is reversed, -2%.) The relationships of school environment, 
class size and school size to achievement are not statistically significant when the within- 
state standardized scores are used, but the relationships are statistically significant when 
the NAEP-adjusted achievement scores are used. Therefore, it is clear that the use of 
NAEP data to incorporate between-state variation, as well as within-state variation, into 
the SASS student achievement score increases the power of the student achievement 
analyses using SASS. 

Comparing Tables 9 and 10, one finds that the adjusted are slightly higher 
when the correlation factor is omitted, for all school types except secondary schools. The 
difference is so minor that there are virtually no differences as a result of including the 
correlation factor. 
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Table 9 — OLS estimate of student achievement (with NAEP mean, standard deviation, and 
correlation) 





Elementary 


Middle 


Secondary 


Combined 


Student Characteristics 


0.475 


0.396 


0.315 


0.357 




(18.762) 


(10.139) 


(8.541) 


(6.884) 


Teacher Qualification 


0.094 


0.103 


0.144 


0.209 


(Rep>orted teaching in 


(3.611) 


(2.868) 


(3.988) 


(4.176) 


Field) 










Teacher Qualification 


0.133 


0.061 


-0.137 


0.002 


(Teacher salary) 


(5.067) 


(1.629) 


(-3.842) 


(0.032) 


School Environment 


0.214 


0.224 


0.280 


0.140 




(8.283) 


(5.263) 


(7.182) 


(2.698) 


Teacher Control 


-0.042 


-0.122 


-0.006 


-0.092 




(-1.664) 


(-3.714) 


(-0.159) 


(-1.872) 


Principals’ Perception 


0.018 


0.135 


-0.038 


0.190 


of Teacher Control 


(0.550) 


(2.711) 


(-0.884) 


(3.102) 


Principal’s Perception 


-0.042 


-0.037 


0.030 


-0.019 


of Own Control 


(-1.174) 


(-0.742) 


(0.712) 


(-0.320) 


Class Size Factor 


-0.167 


-0.263 


-0.250 


-0.334 




(-6.727) 


(-6.458) 


(-6.036) 


(-5.795) 


School Size 


0.004 


-0.104 


0.172 


0.279 




(0.172) 


(-2.422) 


(4.230) 


(4.740) 


Sample Size 


1,096 


515 


609 


355 


Adjusted R^ 


0.4101 


0.3637 


0.3517 


0.3179 



Note; Entries are standardized regression coefficients. Values of Student's t are shown in parentheses. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, 1993-94 Schools and Staffing 



Survey (Public School Teacher Questionnaire, Principal Questionnaire, and School Questionnaire); 1994 NAEP Trial 
State Assessment in reading; state assessment files for 19 states; and data on SAT scores of incoming freshmen at U.S. 
colleges and universities obtained fh>m the Higher Education Research Institute at the Graduate School of Education, 
University of California at Los Angeles. 



o 

ERIC 



42 



58 



Table 10 — OLS estimate of student achievement (with NAEP mean and standard deviation) 





Elementary 


Middle 


Secondary 


Combined 


Student Characteristics 


0.472 


0.402 


0.305 


0.388 




(18.830) 


(10.363) 


(8.230) 


(7.520) 


Teacher Qualifications 


0.074 


0.113 


0.134 


0.175 


(Reported Teaching in 


(2.860) 


(3.161) 


(3.693) 


(3.502) 


Field) 










Teacher Qualifications 


0.137 


0.053 


-0.114 


-0.020 


(Teacher Salary) 


(5.281) 


(1.426) 


(-3.190) 


(-0.393) 


School Environment 


0.244 


0.266 


0.315 


0.161 




(9.517) 


(6.272) 


(8.031) 


(3.113) 


Teacher Control 


-0.023 


-0.119 


-0.014 


-0.098 




(-0.949) 


(-3.105) 


(-0.368) 


(-1.984) 


Principal’s Perception 


0.044 


0.133 


-0.028 


0.184 


of Teacher Control 


(1.230) 


(2.698) 


(-0.651) 


(3.022) 


Principal’s Perception 


-0.054 


-0.040 


0.034 


-0.022 


of Own Control 


(-1.523) 


(-0.806) 


(0.800) 


(-0.372) 


Class Size Factor 


-0.127 


-0.222 


-0.209 


-0.258 




(-5.165) 


(-5.506) 


(-5.033) 


(-4.495) 


School Size 


0.009 


-0.070 


0.190 


0.244 




(0.355) 


(-1.638) 


(4.655) 


(4.162) 


Sample Size 


1,096 


515 


609 


355 


Adjusted R^ 


0.421 1 


0.3720 


0.3442 


0.3226 



Note: Entries are standardized regression coefficients. Values of Student’s t are shown in parentheses. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, 1993-94 Schools and Staffing 
Survey (Public School Teacher Questionnaire, Principal Questionnaire, and School Questionnaire); 1994 NAEP Trial 
State Assessment in reading; state assessment files for 19 states; and data on SAT scores of incoming freshmen at U.S. 
colleges and universities obtained from the Higher Education Research Institute at the Graduate School of Education, 
University of California at Los Angeles. 
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Table 11 — OLS estimate of standardized student achievement scores 



Elementary 


Middle 


Secondary 


Combined 


Student Characteristics 


0.457 


0.402 


0.266 


0.194 




(16.948) 


(10.064) 


(7.025) 


(3.698) 


Teacher (Ratifications 


0.029 


0.055 


0.183 


0.060 


(Reported Teaching in 


(0.671) 


(1.573) 


(4.659) 


(1.268) 


Field) 










Teacher (Ratifications 


0.050 


0.010 


-0.143 


0.063 


(Teacher Salary) 


(1.173) 


(0.293) 


(-3.797) 


(1.340) 


School Environment 


0.257 


0.340 


0.337 


0.296 




(9.310) 


(8.317) 


(8.562) 


(5.869) 


Teacher Control 


-0.028 


-0.082 


-0.035 


-0.026 




(-1.040) 


(-2.220) 


(-0.916) 


(-0.538) 


Principal’s Perception 


0.048 


0.095 


-0.060 


0.042 


of Teacher Control 


(1.422) 


(2.164) 


(-1.459) 


(0.735) 


Principal’s Perception 


-0.042 


-0.016 


0.077 


0.095 


of Own Control 


(-1.266) 


(-0.363) 


(1.874) 


(1.703) 


Class Size Factor 


-0.042 


-0.053 


-0.034 


0.217 




(-1.617) 


(-1.392) 


(-0.805) 


(3.559) 


School Size 


0.044 


-0.019 


0.160 


0.033 



(1.687) (-0,521) (3.985) (0.559) 



Sample Size 


1,096 


515 


609 


355 


Adjusted R^ 


0.3653 


0.3867 


0.2938 


0.2530 



Note: Entries are standardized regression coefficients. Values of Student’s t are shown in parentheses. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, 1993-94 Schools and Staffing 
Survey (Public School Teacher Questionnaire, Principal Questionnaire, and School Questionnaire); state assessment 
files for 19 states; and data on SAT scores of incoming freshmen at U.S. colleges and universities obtained from the 
Higher Education Research Institute at the Graduate School of Education, University of California at Los Angeles. 



5. Can between-state variation in student achievement be extrapolated from 
elementaiy level scores to secondaiy level scores? 



The NAEP 4th grade sample was used to construct achievement scores that 
account for between-state variation. That is, the between-state variation component for 
all schools was set to the value determined for 4th grade. Will the model, therefore, 
predict achievement in elementary schools better than in secondary schools? 

Table 9 and Table 1 1 show the results of regressing NAEP-adjusted achievement 
scores and standardized achievement scores on SASS measures of student characteristics 
and school attributes. The differences in between these two tables indicate the 
contribution of the between-state NAEP-based scale adjustment. The adjusted 
increases from 0.3653 to 0.4101 for elementary schools when NAEP-adjusted 
achievement scores rather than standardized state assessment scores are used. For 
secondary schools, the adjusted R^ increases from 0.2938 to 0.3517. The similarity of the 
gains at the elementary and secondary levels suggests that between-state variation in 
average student reading achievement can be extrapolated from the elementary level to the 
secondary level, for the purposes of exploring school effectiveness using SASS. 



i 



Conclusion 

i 









^ Summary of Findings 

The present study sought to evaluate the feasibility and value of adding school- 
level state assessment data to the SASS database. The principal goals of the study were 
to establish whether existing state files of assessment data can easily be acquired and 
matched to SASS public school records and whether aggregation of analyses across 

® different state assessments can be carried out in a valid manner. To address these issues, 

1993-94 reading assessment data from 19 states, collected previously as part of another 
project, were matched to 1993-94 SASS public schools. Because the Common Core of 
Data contains dual identification codes for public schools (one set for national use, 
including SASS public school sampling, and another set consisting of the various state- 

• assigned school codes), the process of matching was feasible, although about one-third of 
public schools do not have state assessment scores. Average reading achievement scores 
were identified for a total of 2,575 SASS public schools in 19 states. To remove 
irrelevant scaling differences between states, state assessment scores were standardized in 
each state. Then, to reflect between-state variation in achievement. State NAEP reading 

• assessment data were used for state-level means and standard deviations. Correlations of 
school means between state reading assessments and State NAEP were examined to 
refine the resulting SASS achievement measure. 

Once SASS and state assessment data had been merged, a series of analyses were 

• carried out to evaluate the pilot student achievement subfile and explore the value the 
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addition of student achievement data would have for SASS. These analyses addressed 
five research questions. 

How close are the (NAEP-transformed) SASS student achievement scores to 
NAEP school means? The correlations of state assessment school means and NAEP 
school means were computed for states included in the pilot SASS student achievement 
subfile, based on the State NAEP samples in those states and, separately, based on an 
overlap sample of 208 schools that participated in both SASS and State NAEP in these 
states in 1993-94. Most of the resulting correlations were greater than 0.70, suggesting 
that the validity of the SASS student achievement measure as an indicator of school 
reading performance is sufficient for research use. 

How similar are SASS public schools with student achievement scores to other 
SASS public schools? Statistical comparisons, on approximately 50 SASS school 
measures, were made (1) between SASS public schools in the 19 states included in the 
pilot SASS student achievement subfile and SASS public schools in other jurisdictions, 
and (2) between SASS public schools included in the SASS student achievement subfile 
and other SASS public schools in the same states. Although statistically significant 
differences were found, they were not large, suggesting that while inclusion of 
achievement data from a larger number of states may be desirable to increase the 
representativeness of the data set, generalizations based on the pilot SASS student 
achievement subfile can be made meaningfully. 

Are there statistically meaningful school-level relations between the student 
achievement measure and SASS measures? A major aim of the study was to evaluate 
whether meaningful inferences could be made at the school level by examining the 
relations between SASS measures and student achievement measures. A preliminary 
model was developed relating average student achievement in a school to student 
background measures, the school climate, teacher qualifications, and organizational 
factors such as class sizes, school size, and principal and teacher control over school 
operations. Estimates of “path coefficients” based on this model were compared across 
elementary, intermediate, and high schools. Preliminary results indicated that statistically 
significant and meaningful inferences can indeed be made from analyses of the school- 
level correlates of student achievement obtained through SASS. These inferences based 
on SASS data may point the way for controlled studies of the effectiveness of educational 
policy options. 

Does incorporating between-state variation into the student achievement score, 
using NAEP data, increase the power of the student achievement measure? Two sets of 
analyses based on the preliminary model were carried out, the first using the pooled 
(within-state) state assessment measures and the second adding the variation between 
state reading achievement means indicated by State NAEP. SASS school-level measures 
explained a greater percentage of the variation in average student achievement where the 
latter achievement measure was used than where the former measure was used. 



i 



confirming the value of transforming different state assessments onto a common metric 
that includes between-state variance. 

i 

Can between-state variation in student achievement be extrapolated from 
elementary level scores to secondary level scores? Although the State NAEP reading 
scores were only available at the fourth grade level, state assessment scores are available 
in many states for grades ranging from 3 to 1 1. Analyses relating SASS measures to state 
p assessment scores can, therefore, be carried out separately for public elementary, 

intermediate, and high schools. To the extent that between-state variation in foxirth grade 
reading achievement, as measured by State NAEP, is indicative of between-state reading 
achievement variation in later grades, the same NAEP-based adjustment for between-state 
variation can be added to reading scores in all grades. Comparative analyses were carried 
^ out to assess whether this adjustment was accompanied by degradation of the preliminary 

model’s predictive power in the higher grades, but no noticeable degradation was 
observed. 

Value of the SASS Student Achievement Subfile 

The potential value of the SASS student achievement subfile for educational 
policy analysis is enormous. Although it cannot substitute for a controlled experiment in 
attempting to make causal inferences, it can provide a uniquely powerful source of 
evidence on the relations among school factors and average school achievement. It is 
^ based on a sample that is representative of a substantial portion of the nation’s schools 

and is of sufficient size to identify important patterns; it includes a wide range of 
information on school programs, resources, and dynamics; and it includes in-depth 
assessments of teacher and principal characteristics and attitudes about teaching, the 
students, and the school. 

^ The value of the SASS student achievement subfile lies not only in its relevance 

for national-level policy research but also in its relevance for state-level reform efforts. 
Because SASS collects information that encompasses educational practices that state 
reforms are striving to affect, analyses within each state can enhance the interpretation of 
state assessment scores, while at the same time showing how each state relates to a 
^ combination of other states. 

There is also an indirect value to be gained from creating the SASS student 
achievement subfile. By incorporating a state-produced data source into SASS, NCES 
increases the role of state education agencies as constituencies of SASS. To the extent 
^ that each state sees a role for SASS in helping to interpret its state assessment and reform 

results in a national context, the potential for an integrated state-federal school-level data 
system aimed at improving each state’s policy research capacity is fostered. 
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Next Steps 



Based on the positive results of the pilot study, we plan to carry out additional 
activities to complete the SASS student achievement subfile for the 1993-94 survey and 
to publish a report on factors related to student achievement based on analyses of 1993-94 
SASS. In particular, we will: 

• add state assessment data for a few additional states, 

• add state assessment data for 1992 and 1996, for some states, and 

• add state mathematics assessment scores to the file. 

We will also investigate various analytical issues related to the use of 
state assessment achievement scores in analyses based on SASS, including: 

• differences in models for reading and for mathematics 
achievement, 

• variation between assessments in different grade levels, 

• relations between average student achievement and other outcome 
indicators, and 

• variation of results between states. 

Finally, we will report on the estimated costs associated with the creation of a 
student assessment subfile in a similar manner for future SASS files and sununarize 
existing information about the expected availability of state student assessment data in the 
future. Ideas for extending the subfile in the future, for example by requesting reported 
average achievement scores in nationally standardized tests fi-om participating private 
schools, will be explored. 

As proposed, the procedures for creating a SASS student achievement subfile 
have no direct implications for changes in the design of SASS. However, two aspects of 
the current design are important: (1) the inclusion of a sufficiently large sample of 
schools in each state to ensure sufficient degrees of freedom for taking into account the 
state assessment mean and standard deviation at each assessed grade and (2) the inclusion 
of a sufficiently large sample of teachers in each participating school to produce school- 
level average estimates for teacher survey responses. Both aspects of SASS add power to 
achievement analyses as well as to other SASS analyses. 
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Appendix A 



Technical Notes 

I. Factor Analysis of Teacher and Principal Attitudinal Items 

A series of attitudinal items were included in the principal’s and teacher’s 
questioimaires for SASS. These items were previously factor analyzed to identify a set of 
factors that would provide more reliable information about teachers, administrators, and 
their schools than individual items. These are basically the same factors that were used in 
the Private Schools in the United States: A Statistical Profile, 1993-94 (McLaughlin and 
Broughman, 1 997). Some of these factors were included in the Student Characteristic, 
School Environment, and Teacher’s Classroom Control Composites used in this project, 
instead of the original variables. They are scaled from 0 to 10 in which 10 indicates the 
least problem and most teacher control. Please note that the items for the principal and 
teacher questioimaires are different. 

The SASS items used in constructing the factors are the following: 

Item 47-a set of 25 items to be rated on a four-point scale from “strongly agree” to 

“strongly disagree,” of which the following 3 were used: 

e. level of student misbehavior 

k. principal enforces school rules for student conduct and backs me up 



m. rules for student behavior are consistently enforced by teachers 



Item 46-a set of 23 items about problems rated on a four-point scale of “not a 
problem, minor, moderate, serious,” of which the following 20 were used: 



a. student tardiness 

c. teacher absenteeism 

e. physical conflicts among students 

g. vandalism of school property 

i. student use of alcohol 

k. student possession of weapons 

m. student disrespect for teachers 

o. student apathy 

q. lack of parental involvement 

s. poverty 



b. student absenteeism 

d. students cutting class 

f robbery theft 

h. student pregnancy 

j. student drug abuse 

1. verbal abuse of teachers 

n. students dropping out 

p. lack of academic challenge 

r. parental alcoholism and drug abuse 

t. racial tension 



Item 44-a set of 6 items on influence on school policies, scaled from 0 (“no 
influence”) to 5 (“a great deal of influence”), of which the following one was 
used: 



f. establishing curriculum 

Item 45-a set of 6 items on control of classroom practices, scaled fi’om 0 (“no 
control”) to 5 (“complete control”): 

a. selecting textbooks and other instructional materials 

b. selecting context, topics, and skills to be taught 

c. selecting teaching techniques 



d. evaluating and grading students 



e. discipline students 

f. determining the amount of homework to be assigned 

Item 24-a set of 2 items from principal questionnaire on school problems, scaled 
from 1 (“serious”) to 4 (“not a problem”): 

p. lack of academic challenge 

q. lack of parent involvement 

Eleven factors were identified, the first of which was included in the student 
characteristics composite, the next eight of which were included in the school climate 
composite, and the last two of which were included as an organizational composite: 

Community: Lack of serious problems stemming from parental alcoholism/drug abuse, 
poverty, and racial tension. Item 46 - (r, s, t) 

Basic Standards: Lack of serious problems with robbery, drug abuse, alcohol, pregnancy, 
and dropouts. Item 46 - (f, h, i, j, n) 

Attendance: Lack of serious tardiness and absence problems. Item 46 - (a, b, c, d) 

Respect for Teachers: Lack of serious problems with verbal abuse and disrespect of 
teachers. Item 46 - (e, k, 1, m) 

Respect for Students: Belief that students are capable of learning and do not have 
attitudes and habits that reduce their learning potential. Item 47 - (e) 

Apathy: Item 47 (p,q) 

Respect for Property: Lack of vandalism and theft (factor for principals only). Item 46 - 
(f,g) 

Behavior Standards: Systematic enforcement of agreed upon discipline policy and 
student behavioral standards. Item 47 - (k, m) 

Satisfaction: Choose to become teacher again. Item 48 

Teachers ’ Influence on Curriculum: Item 44 - (f). Item 45 - (a, b) 

Teachers ’ Classroom Control: Selecting teaching techniques, assigning homework, and 
evaluating and disciplining students. Item 45 - (c, d, e, f) 



II. College quality data: SAT scores 

The data file underlying the estimates of college quality was obtained from the 
Higher Education Research Institute at the Graduate School of Education, University of 
California at Los Angeles. The item used is the SAT score for incoming students for the 
years 1972, 1978, and 1982. These data were obtained in electronic form for 
approximately 2,300 colleges in the United States. The procedures for using the SAT 
score database were as follows: 

(1) The SAT score database was matched to the 1982-83 HEGIS (IPEDS 
predecessor) dataset by FICE code identifiers to pick up college names. 

(2) The database was then matched to the 1987-88, 1990-91, or 1993-94 IPEDS 
dataset by FICE code to check for consistency of identifying codes through the 
years (FICE code identifiers change as colleges lose and gain accreditation). 

(3) Finally, the database was matched to individual teacher records on SASS by 
IPEDS identification codes. 



i 



III. CCD variables and state file variables used to merge SASS with state 
assessment data 



State 


Primary 
CCD variables 


Primary 
state variables 


Secondary 
state variables 


Alabama 


sc_city, sc_name 


city (12), sch name (18) 




California 


sc_stid (7), sc_stsid (7)” 


sch_code (14) 


sch_name 


Delaware 


sc_stid (5) 


distcode (2), sch_code (3) 


sch_name 


Florida 


sc stid (2), sc_stsid (4) 


distcode (2), sch_code (4) 


sch_name 


Georgia 


sc stid (3), sc_stsid (4) 


syscode (3), sch_code (4) 


sch_name 


Hawaii 


sc_stsid (3-7) 


sch_code (5) 


sch_name 


Kentucky 


scstsid (4-12) 


schcode (9) 


distname, sch name 


Louisiana 


scstsid (6) 


schcode (6) 


No 


Maine 


scstsid (5) 


sch_code (5) 


sch_name, distcode 


Massachusetts 


scstsid (6) 


sch_code (6) 


sch_name 


Michigan 


sc stid (5), sc stsid (4) 


distcode (5), sch code (4) 


sch_name, distname 


Montana 


sc_stid (10-13) 


le(4) 


district (sch name) 


New York 


sc_stsid (12) 


sch_code(12) 


None 


Pennsylvania 


scstsid (13) 


distcode (9), sch_code (4) 


distname, sch_name 


Rhode Island‘d 




s_city (15), sch_name (3 1) 


distname, sch_name 


Tennessee 


00, sc stid (3), sc stsid (4) 


syscode (5), sch_code (5) 


sch_name, city 


Texas 


scstsid (9) 


sch_code (campus) (9) 


schname 


Utah 


sc stid (2), sc stsid (4-6) 


distcode (2), sch code (3) 


sch_name 


Washington 


sc stid (S), sc stsid (4) 


distcode (5), sch code(bldg) (4) 


sch name, distname 



Note: Numbers in parentheses indicate the number of characters used in the particular code field. For % 

example, in Florida, the district code has two digits and the school code has four digits. Names for state 
variables have been regularized across states for presentation in this table. 



7 digits of the CCD State Agency ID and 7 digits of the CCD State School ID were combined to merge 
with the 14 digits sch_code from the state assessment file in California. Different numbers of digits were 
used in different states 

We did not use CCD variables for Rhode Island because we merged the SASS schools with the state 
assessment schools in a booklet. 
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